According to @_avichawla on Twitter, attention workloads in transformers repeatedly shuttle large matrices between on-chip SRAM and high bandwidth memory to compute QK products and softmax, which creates significant memory bandwidth pressure across layers. As reported by the tweet thread, Q and K matrices are distributed to threads for parallel compute, with the QK product written back to HBM; the softmax stage similarly redistributes the product to threads, computes, and writes outputs to HBM, then repeats per layer. According to this description, the bottleneck implies business opportunities for kernel-level optimizations like FlashAttention, fused attention, and recompute-aware tiling, as well as hardware strategies such as larger SRAM, better tensor core utilization, and near-memory compute. As noted by the source, the repeated SRAM-HBM traffic underscores why IO-aware attention kernels, KV cache compression, and sequence-parallelism are key levers for reducing latency and cost in LLM serving and training.

According to Jeff Dean on X, TPU v8i is co-designed with Google’s Gemini research team to deliver low-latency inference by incorporating large on-chip SRAM that reduces trips to HBM for model weights and KV cache state, enabling more computations to stay on chip. As reported by Jeff Dean, these memory locality improvements target transformer serving bottlenecks—specifically attention KV cache bandwidth and latency—helping accelerate token generation and lower tail latency in LLM inference. According to Jeff Dean, the design focus implies better cost efficiency for enterprise-scale Gemini deployments, higher throughput per watt, and improved responsiveness for real-time applications such as chat, code assistance, and multimodal agents.

According to Avi Chawla (@_avichawla), large language model inference differs fundamentally from traditional ML because output is generated token-by-token via hundreds of sequential forward passes, making prefill compute-bound and decode memory-bandwidth-bound, which degrades performance when co-located on the same GPU (as reported by his X post and linked article). According to Chawla, KV cache size grows with conversation length and is shared across requests, shifting routing from least-busy to prefix-aware replica selection, while Mixture-of-Experts introduces expert parallelism not seen in classic serving (as reported on X). According to Chawla, these constraints birthed a new optimization stack spanning nine pillars—compression, attention, KV cache management, batching, decoding, parallelism, routing, plus production-specific scheduling and memory optimizations—mapping 72 concrete techniques for production LLMs (as reported by his X article summary). Business impact: according to Chawla, operators can cut latency and GPU spend by separating prefill/decode placement, using prefix-aware routing, cache eviction policies, paged KV memory, speculative decoding, and MoE-aware load balancing—key levers for cost per token, throughput, and user latency SLAs in 2026 LLM deployments.

According to AndrewYNg on X, DeepLearning.AI launched a new course, Efficient Inference with SGLang: Text and Image Generation, created with LMSys and RadixArk and taught by Richard Chen of RadixArk. As reported by AndrewYNg, the course targets production LLM cost bottlenecks and latency using SGLang techniques such as kernel fusion, paged attention, continuous batching, and optimized KV cache management for both text and image generation. According to AndrewYNg, the curriculum emphasizes practical deployment patterns for serving large models at scale, highlighting business value through reduced GPU hours, higher throughput per dollar, and improved tail latency—key metrics for inference economics.

According to DeepLearningAI on Twitter, a new course titled Efficient Inference with SGLang: Text and Image Generation is now live, focusing on cutting LLM inference costs by eliminating redundant computation using KV cache and RadixAttention (source: DeepLearning.AI tweet on April 8, 2026). As reported by DeepLearning.AI, the curriculum demonstrates how SGLang accelerates both text and image generation by reusing key value states to reduce recomputation and applying RadixAttention to optimize attention paths for lower latency and memory usage. According to DeepLearning.AI, the course also translates these techniques to vision and diffusion-style workloads, indicating practical deployment benefits such as higher throughput per GPU and reduced serving costs for production inference. As reported by DeepLearning.AI, the material targets practitioners aiming to improve utilization on commodity GPUs and scale serving capacity without proportional hardware spend.